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PREFACE 


This  report  documents  an  experiment  using  an  eye  tracking  system  to  analyze 
pilots'  scan  paths.  This  research  was  conducted  under  Work  Unit  1 123-B2-06,  Aircrew 
Training  Research  Support;  Contract  F41624-95-C-5011  with  Hughes  Training,  Inc- 
Training  Operations.  The  Principal  Investigator  was  Dr  Byron  J.  Pierce;  the  Laboratory 
Contract  Monitor  was  Mr  Daniel  Mudd. 

This  effort  is  part  of  an  Armstrong  Laboratory,  Human  Resources  Directorate, 
Aircrew  Training  Research  (AL/HRA)  program  to  provide  behavioral  research  support 
for  the  Air  Force's  Warfighter  Training  Effectiveness  Behavioral  Research  which 
includes  eye  tracking  technology  for  aircrew  training. 
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An  Eye  Tracking  System  for  Analysis  of  Pilots’  Scan  Paths 

INTRODUCTION 

Since  the  early  1900s,  a  variety  of  objective  methods  have  been  developed  for  the  purpose  of 
measuring  eye  position.  These  devices  have  been  instrumental  in  evaluating  basic  motor  and 
neurological  aspects  of  the  oculomotor  control  system,  as  well  as  in  providing  diagnostic  information  for 
medical  and  clinical  evaluations.  Numerous  behavioral  and  human  factors  studies  have  been  conducted 
using  these  eye-tracking  technologies  to  measure  eye  position.  Information  gathered  using  these  systems 
has  been  integral  in  determining  field-of-view  (FOV)  requirements  in  simulators  (Dixon,  Rojas,  Krueger,  & 
Simcik,  1990),  and  in  controlling  high  resolution  area-of-interest  display  systems  (Wetzel,  Thomas,  & 
Williams,  1 990).  The  predecessor  of  the  current  portable  system  described  herein  was  used  to  assist  in 
the  evaluation  of  situational  awareness  and  as  a  post-mission  debriefing  tool  in  an  F-15  simulator  at 
Armstrong  Laboratory's  Aircrew  Training  Research  Division,  in  Mesa,  AZ,  using  mission-ready  F-15  pilots. 
This  system  used  a  small  video  camera  and  an  eye  tracker  attached  to  a  helmet  worn  by  the  pilot.  An 
electronic  cursor  corresponding  to  eye  position  was  then  generated  and  combined  with  video  from  the 
scene  camera.  Using  this  system,  we  were  able  to  accurately  determine  the  pilots'  line  of  sight  (direction 
of  gaze)  to  any  instrument  in  the  cockpit.  In  addition,  we  were  also  able  to  confirm  visual  contact  with  out- 
of-the-cockpit  objects.  These  data,  along  with  all  cockpit  communications,  were  then  recorded  onto  video 
tape  for  manual  analysis.  In  the  debriefing  sessions  subject-matter  experts  were,  on  occasion,  able  to 
determine  and  show  the  pilot  occurrences  of  missed  cues  or  other  events  that  led  to  reduced  pilot 
situational  awareness,  and  in  some  cases,  reduced  mission  effectiveness.  Manual  analysis  of  the  video 
tapes  allowed  us  to  determine  the  scan  path  sequence,  dwell  time,  frequency  . of  instrument  usage,  and 
overall  situational  awareness  from  selected  pilots. 

In  the  training  environment,  eye  tracking  can  be  used  to  differentiate  and  objectively  describe  visual 
behavior  differences  between  "expert"  and  “novice”,  or  less  skilled,  pilots  or  system  operators  revealing 
effective  training  techniques  and/or  eye  movement  behavior  that  may  be  difficult  to  verbally  describe, 
otherwise  observe,  or  diagnose  by  others.  Further,  line-of-sight  information  can  also  be  used  to  provide 
feed-back  to  the  user  with  respect  to  scanning  technique,  detection  of  scan  breakdown,  and  as  a  teaching 
and  diagnostic  tool  for  improving  training  by  showing  effective  eye  movement  behavior. 

In  this  report,  we  describe  the  development  of  a  lightweight,  easy-to-use,  portable  eye  position 
measurement  system  and  a  companion  automated  analysis  system.  We  also  discuss  several  training 
related  applications  for  which  the  portable  system  has  been  used.  To  date,  this  system  has  been  used 
with  T-37  instructor  pilots  (IPs),  T-37  student  IPs  (rated  pilots  training  to  be  IPs),  and  F-16  IPs  in  three 
aircraft  simulators:  (a)  The  T-37  motion  base  simulator  at  Randolph  AFB,  TX;  (b)  The  F-16  Air  Combat 
Engagement  Simulator  (ACES)  (formerly  the  Simulator  for  Air-to-Air  Combat,  or  SAAC)  at  Luke  AFB,  AZ; 
and  (c)  The  F-16  LANTIRN  simulator,  also  located  at  Luke  AFB. 

A  Basis  For  Eye  Movements 

We  move  our  eyes  primarily  to  extract  information  from  the  visual  environment  and  to  stabilize  images 
on  the  retina  during  free  head  movement.  Our  eyes  also  move  to  compensate  for  changes  in  head 
position  and  orientation  as  sensed  by  the  vestibular  system.  The  vestibular  system  is  also  responsible  for 
changing  eye  position  based  on  gravitational  and  accelerative  input  forces.  Because  the  world  is 
perceived  as  stable  and  visually  clear  throughout  the  visual  field,  we  are  often  unaware  of  our  own  eye 
movements  or  of  the  integrating  processes  that  occur.  Nevertheless,  eye  movements  and  scanning 
patterns  are  indicators  of  thought  and  mental  processing  involved  during  visual  information  extraction. 

We  move  our  eyes  almost  constantly,  extracting  information  from  the  visual  world.  Due  to 
organization  of  the  eye,  specifically,  the  retina,  the  clarity  with  which  we  see  an  object  and  the  extent  to 
which  it  is  resolved  is  dependent  on  the  position  of  our  eyes  with  respect  to  the  object  of  interest.  Our 
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ability  to  see  color  and  fine  detail  is  limited  to  a  very  small  centrally  located  area  on  the  retina  called  the 
fovea  which  is  roughly  1.5  millimeters  in  diameter.  Consequently,  one  of  the  basic  functions  of  the  eye 
movement  control  system  is  to  position  and  stabilize  the  eyes  such  that  the  projected  image  of  an  object 
falls  on  the  fovea,  allowing  maximum  extraction  of  visual  information  concerning  that  object.  An  example 
of  head  and  eye  movement  occurring  during  visual  scanning  is  shown  in  Figure  1. 


Figure  1 .  Examples  of  Saccadic,  Compensatory  Smooth  Eye  Movement  and 
Head  Movement  During  Visual  Scanning.  For  clarity,  only  the  horizontal 
component  of  head  and  eye  movement  and  resulting  gaze  are  shown.  The  sum 
of  head  and  eye  position  is  equal  to  the  line  of  sight  or  gaze  position.  Stabilized 
gaze  results  from  the  eyes  moving  at  roughly  equal  rate  but  opposite  in  direction 
to  that  of  head  during  the  stabilized  fixation  period. 

Fast  eye  movements  called  saccades,  occur  during  visual  search,  while  reading,  or  when 
performing  instrument  cross-checks.  Saccades  are  volitional  and  during  the  ballistic  portion,  the  eyes  can 
reach  peak  velocities  in  excess  of  400®/s,  reaching  peak  accelerations  in  excess  of  30,000°/s2,  all  in  under 
50  ms.  Between  saccadic  movements,  the  eyes  pause  and  fixate  for  at  least  200  ms.  We  can  voluntarily 
change  our  eye  position  no  more  than  4  to  5  times  per  second.  We  can  accurately  position  our  eyes  on  a 
target  within  a  couple  tenths  of  a  degree  from  the  retinal  fovea.  During  the  pause  when  the  object  is 
stabilized  on  the  fovea,  the  maximum  amount  of  visual  information  about  the  object  or  target  is  extracted. 
Free  head  movement  during  visual  search,  as  shown  in  Figure  1,  allows  the  eyes  to  move  at  roughly 
equal  velocity  but  opposite  direction  to  that  of  the  head  which  stabilizes  an  image  on  the  retina.  If  the 
head  is  stationary  and  a  target  moves  smoothly,  the  eyes  will  rotate  at  roughly  the  velocity  of  the  target. 
Generally,  smooth  eye  movement  is  involuntary;  we  cannot  move  our  eyes  smoothly  in  the  absence  of  a 
target.  If  the  target  and  eye  velocities  match,  in  effect,  the  eye  has  stabilized  on  the  target,  and  the  fine 
details  of  the  target  can  be  realized.  This  eye  movement  system  is  best  at  tracking  smoothly  moving 
targets  whose  velocities  are  often  less  than  30°/s.  Errors  in  tracking  in  excess  of  a  few  tenths  of  a  degree 
are  often  corrected  by  saccades  which  rapidly  reposition  the  eye  back  on  the  target.  If  the  movement  of 
an  object  is  smooth,  but  at  a  higher  velocity,  the  ability  to  maintain  smooth  eye  movement  diminishes. 

THE  PORTABLE  EYE  MEASUREMENT  SYSTEM 

The  ability  to  measure  eye  position  provides  a  tool  for  knowing  the  look  point  of  that  person.  Most  people, 
however,  are  not  always  consciously  aware  of  where  their  eyes  are  looking  at  any  given  instant  of  time. 
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The  measurement  of  eye  position,  therefore,  provides  a  detailed  record  and  history  of  visual  behavior  and 
scanning  patterns  over  time. 

A  number  of  devices  and  techniques  have  been  developed,  such  as  electro-oculography,  magnetic 
search  coil  systems,  photoreflective  methods,  imaging  systems,  and  video-based  image  devices  to  obtain 
accurate  measurements  of  eye  position.  Due  to  the  nature  of  these  types  of  devices  and  their  limitations, 
most  have  been  restricted  to  use  in  the  laboratory.  Fewer  systems  have  been  specifically  developed  for 
non-laboratory  or  “non-stationary”  conditions  and  are  able  to  provide  accurate  information  for  various 
applications.  These  systems  track  a  combination  of  an  estimate  of  the  pupil  center  and  corneal  reflections 
to  account  for  helmet  slippage  in  a  dynamic  environment.  Still,  major  limitations  of  systems  used  in 
simulator/eye-  tracking  studies  include  transporting  the  bulky,  yet  fragile,  system  components,  difficulty  of 
operation,  and  the  amount  of  time  and  labor  required  to  analyze  and  compile  the  data.  The  manual 
analysis  process  could  often  take  days  to  analyze  only  a  few  minutes  of  eye  position  data. 

The  positive  response  from  subjects,  including  the  pilots  from  the  F-15  situational  awareness  test  and 
subsequent  off-site  tests  in  a  commercial  Boeing  757  simulator,  encouraged  us  to  pursue  development  of 
the  current  companion  systems.  The  primary  goals  of  this  effort  have  been  to  reduce  the  size  and  weight 
of  the  system,  while  improving  overall  ease  of  use,  setup  time,  and  accuracy.  Further,  decreasing  the 
time  and  effort  required  to  analyze  the  data  was  also  key.  Based  on  information  gathered  from  pilots,  the 
head-mounted  eye  tracker  and  video  scene  camera  were  redesigned  to  increase  comfort  and  reduce 
visual  restrictions.  The  video-based  automated  analysis  system,  described  herein,  allows  rapid 
quantification  of  eye  movement  scan  path  and  other  visual  information.  The  system  reduces  the  analysis 
and  data  reduction  process  to  almost  real  time.  The  portable  eye-tracking  system  and  companion 
automated  analysis  system  have  been  under  development  for  the  past  several  years  in  close  collaboration 
with  El  Mar,  Inc.  and  Armstrong  Laboratory  in  Mesa,  AZ, 

Features  of  the  Portable  Eye  Measurement  System 

The  complete  portable  eye  measurement  system  is  shown  in  Figure  2.  The  system  consists  of  a  main 
processor  unit,  a  hand  controller,  an  8  mm  VCR  with  recorder  and  display,  and  an  adjustable  headband 
supporting  the  eye  tracker  sensor  and  scene  camera.  The  headband  also  supports  a  standard  military- 
style  adjustable  microphone  and  a  pair  of  miniature  earphones. 


Figure  2.  Components  of  the  El  Mar,  Inc.  Vision  2000  Portable  Eye 
Measurement  System.  From  left  to  right,  the  8  mm  VCR  and  display,  headband 
with  eye  tracker  sensor  and  side-mounted  scene  camera,  the  main  processor, 
and  hand  controller  unit. 
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The  weight  of  the  main  processing  unit  is  approximately  8  kg.  The  headband  with  scene  camera  and 
communications  system  weighs  approximately  300  g.  The  complete  system  can  be  transported  within  a 
single  case. 

Principle  of  Eye  Measurement  Operation 

The  system  is  based  on  the  dark  pupil  principle.  It  provides  estimates  of  horizontal  and  vertical  eye 
position  and  pupillary  size  at  a  rate  of  60  Hz. 

The  system  uses  an  estimate  of  pupil  center  and  compares  this  measurement  with  the  positions  of  two 
corneal  reflections.  The  virtual  images  formed  by  the  corneal  reflections  are  the  result  of  two  noncolli- 
mated  infrared  light  emitting  diodes  (LEDs)  located  within  the  plastic  housing  above  the  eye.  The  infrared 
LEDs  are  pulsed  at  60  Hz  and  homogeneously  illuminate  the  eye  with  less  than  300  microwatts/cm^  of 
energy.  Because  the  infrared  sources  and  the  imaging  optics  are  not  coaxial  to  the  eye,  the  pupil  remains 
dark.  A  shatterproof  plastic  beam-splitter  located  below  the  line  of  sight  of  the  right  eye  redirects  the 
reflected  infrared  (IR)  light  from  the  eye  to  a  two-dimensional  charge-coupled  device  (CCD)  array  located 
in  the  plastic  housing  above  the  eye.  A  set  of  knobs  on  the  plastic  housing  allows  the  user  to  focus  and 
translate  the  image  of  the  eye. 

Figure  3  exhibits  an  infrared  image  of  the  eye  showing  the  two  corneal  reflections  (two  bright  dots)  and 
an  estimate  of  the  dark  pupil  center  (the  crosshair).  The  system  can  measure  eye  movements  up  to  ±45° 
horizontal  (H)  and  up  to  ±35°  vertical  (V).  The  system  has  an  accuracy  of  better  than  ±0.5°  for  changes  in 
eye  position  less  than  ±15°  H  and  V  which  decreases  to  ±1°  at  the  limits.  The  resolution  of  the  system  is 
better  than  0.2°.  The  pupillary  corneal  difference  method  also  provides  a  method  to  distinguish  between 
headband  movement  and  changes  in  eye  position.  Without  this  compensation,  changes  in  headband 
position  of  only  1  mm  would  appear  equivalent  to  an  eye  movement  of  up  to  10°.  Wth  the  portable 
system,  translations  up  to  5  mm  can  be  tolerated. 


Figure  3.  Infrared  Image  of  the  Eye  Taken  from  the  Portable  System,  Showing 
Two  Corneal  Bright  Reflections  (Surrounded  By  Dotted  Boxes)  and  the  Pupillary 
Center  Estimate  Indicated  by  the  Cross.  Eye  position  is  based  on  the  difference 
between  the  pupillary  center  estimate  and  the  positions  of  the  corneal 
reflections. 
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The  Main  Processor  Unit. 


The  main  processor  unit  is  a  stand-alone  device,  controlling  all  functions  of  the  eye-tracking  system. 
The  primary  control  functions  include;  eye  position  and  pupillary  estimation,  eye  calibration,  video  display, 
cursor  generation  and  control,  and  connections  to  a  VCR  and  PC.  All  functions  of  the  main  processor  unit 
can  be  accessed  and  controlled  via  the  hand-held  key  pad.  Program  and  eye  tracking  parameters  are 
stored  in  electrically  erasable  programmable  ROM.  Program  updates  can  be  loaded  from  any  DOS-based 
PC  to  the  main  processor  unit  via  one  of  two  serial  ports.  A  serial  port  can  also  be  used  to  graphically 
view  actual  eye  movement  data  via  connection  to  a  PC.  Using  the  hand-held  key  pad,  various  system  and 
control  parameters  can  be  set  for  a  specific  training  environment  or  condition,  such  as  calibration  angles, 
scene  camera  focal  length,  and  cursor  style.  Data  from  the  portable  system  are  recorded  onto  a  high 
fidelity  stereo  VCR  tape  recorder.  These  data  include  video  from  the  scene  camera  and  cursor,  as  well  as 
encoded  audio  data  comprising  horizontal  and  vertical  eye  position,  pupillary  size,  blink  and  system  status 
information,  and  frame  count.  The  other  audio  channel  can  be  used  to  record  voice  communications.  Eye 
position  data  are  stored  at  a  rate  of  60  Hz.  Additional  stored  data  include  the  recording  of  event/playback 
markers,  system  status,  and  frame  counter  information.  The  hand-held  key  pad  allows  the  user  to  mark 
"significant  events"  on  the  video  tape  for  later  review  without  having  to  search  the  entire  tape. 

The  hand  controller  key  pad  controls  the  basic  functions  of  the  portable  system.  When  preparing  to 
eye  track  a  subject,  the  key  pad  is  used  to  select  and  control  various  functions  of  the  main  processor  unit 
required  during  the  initial  adjustment  and  calibration  of  the  eye  measurement  system.  With  the  eye 
tracker  headband  on  the  individual,  the  key  pad  can  be  used  to  switch  between  an  image  of  the  eye,  as 
shown  in  Figure  3,  and  imagery  from  the  head-mounted  scene  camera.  If  the  eye  alignment  is  correct,  a 
button  on  the  key  pad  can  be  pressed  which  initiates  a  startup  procedure,  automatically  adjusting  the 
infrared  LED  intensities  and  contrast  for  optimal  performance  based  on  characteristics  of  the  individual’s 
eye.  This  process  takes  less  than  five  seconds.  Upon  completion  of  the  automated  startup  process,  a 
message  is  displayed  on  the  screen  confirming  the  results  of  the  adjustment.  The  system  also  allows  the 
user  to  manually  override  or  adjust  the  LED  intensities  and  contrast  via  the  key  pad  should  special 
circumstances  or  conditions  require  manual  adjustment. 

The  key  pad  is  used  during  calibration  of  the  eye  measurement  system.  During  calibration,  the 
individual  is  asked  to  look  at  a  series  of  known  calibration  points  placed  along  the  horizontal  and  vertical 
axes  on  a  calibration  card.  At  each  point,  a  measurement  of  eye  position  is  taken  by  pressing  a  button  on 
the  key  pad.  At  the  conclusion  of  the  calibration,  a  message  is  sent  to  the  display  screen  confirming  the 
outcome  of  the  calibration.  The  scene  camera  is  then  aligned  with  respect  to  the  position  of  the 
individual's  eye.  The  key  pad  is  used  to  properly  align  the  cursor,  corresponding  to  eye  position,  with 
respect  to  the  head-mounted  scene  camera.  The  key  pad  may  also  be  used  to  make  minor  cursor 
corrections,  if  required. 

The  Recorder  and  Video  Display. 

The  portable  system  includes  a  compact  Sony  Watchman  used  for  eye  and  scene  alignment, 
calibration,  and  for  observation  of  the  user's  gaze  response.  Based  on  our  requirements  to  resolve 
imagery  from  the  F-16  Head-Up  Display  (HUD)  during  playback,  we  use  a  Super  VHS  recorder  which 
offers  nearly  twice  the  resolution  of  a  standard  VHS  recorder.  To  simplify  operator  use,  the  functions  of 
the  recorder  can  be  controlled  via  the  hand-held  key  pad. 

The  Scene  Camera. 

A  black-and-white  scene  camera  (Elmo  Model  EM  1 02-BW)  is  mounted  to  the  side  of  the  headband 
support.  For  most  applications,  the  scene  camera  is  mounted  level  with  the  eyes  at  roughly  the  same 
distance  to  the  objects  of  interest  as  to  that  of  the  eye.  When  used  with  aircraft  having  collimated  HUDs, 
the  scene  camera  must  be  mounted  in  the  pupillary  plane,  directly  between  the  eyes  to  minimize  parallax. 
A  variety  of  interchangeable  lenses  have  been  used  based  on  the  size  of  the  aircraft  instrument  panel,  the 
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level  of  detail  required,  and  scene  camera  FOV.  Presently,  we  have  used  lenses  with  horizontal  FOVs  of 
approximately  1 02°,  90°  and  45°.  Often,  the  choice  of  a  lens  is  determined  by  balancing  the  requirements 
of  FOV  and  detail,  considering  that  as  FOV  increases,  detail  decreases. 

THE  AUTOMATED  ANALYSIS  SYSTEM 

The  automated  analysis  system  performs  image  analysis  on  each  frame  of  the  recorded  video, 
combining  these  data  with  eye  position  coordinate  data  to  determine  the  line  of  sight.  To  date,  the  system 
has  been  used  to  analyze  data  collected  during  research  studies  conducted  with  T-37  IPs  and  T-37 
student  IPs  while  performing  instrument  landings  in  a  T-37  motion  base  simulator  at  Randolph  AFB. 

To  determine  the  line  of  sight  to  an  object,  both  head  and  eye  position  must  be  known.  A  significant 
advantage  of  the  portable  system  is  that  it  does  not  require  the  use  of  an  external  head  position 
measurement  system  which  is  often  prone  to  interference  from  other  objects  or  surrounding  obstructions. 
Instead,  we  estimate  head  position  based  on  a  set  of  unique  reference  markers  placed  within  the  FOV  of 
the  scene  camera. 

Using  a  method  similar  to  triangulation,  head  position  is  estimated  based  on  the  relationship  between  a 
set  of  fixed  reference  markers.  The  reference  markers  resemble  the  spokes  of  a  wagon  wheel  with  each 
having  its  own  unique  radial  pattern.  Using  cross-correlation  techniques,  the  image  analysis  system 
Identifies  and  tracks  the  relationships  between  the  reference  markers  as  the  head  moves.  For  example,  if 
the  head  moves  closer  to  the  instrument  panel,  the  markers  appear  to  have  moved  farther  apart.  If  the 
head  moves  farther  away,  the  markers  appear  to  move  closer  together.  If  the  head  changes  orientation, 
the  position  of  the  markers  appears  to  rotate. 


For  a  6-degree-of-freedom  measurement  of  head  position,  a  minimum  of  three  reference  markers  must 
be  tracked.  For  proper  registration  of  head  position,  a  minimum  of  two  markers  must  be  within  the  FOV  at 
all  times.  Figure  4  shows  a  set  of  reference  markers,  roughly  4  cm  in  diameter,  located  within  the  T-37 
cockpit  instrument  panel. 

A  Pentium-based  PC  performs  the  automated  analyses  of  the  video  tape.  Prior  to  analysis,  a  single¬ 
image  frame  from  the  video  recording  is  grabbed  and  the  objects  of  interest  are  identified  by  placing  boxes 
around  each  instrument  of  interest.  Next,  the  reference  markers  are  identified  and  the  tape  is  played 
through  the  system.  An  additional  set  of  algorithms  is  then  used  to  identify  the  saccadic  eye  movement 
responses,  and  the  times  of  the  fixation  pauses.  Line-of-sight  measurements  which  fall  within  an 
instrument  boundary  are  counted  as  fixations  on  an  instrument,  based  on  eye  movement  criteria  defining 
the  saccadic  response  and  fixation  pause. 

Methods  are  presently  being  developed  to  create  boundaries  which  more  closely  match  the  shape  of 
the  instrument  or  object  and  to  resolve  cases  where  the  line  of  sight  falls  on  a  boundary  edge.  An  overlay 
showing  the  scan  path  of  one  T-37  pilot  radiating  from  the  ADI  can  also  be  seen  in  Figure  4.  usage, 
scanning  The  data  provided  by  the  program  includes  information  on  instrument  sequence,  and  dwell  time. 
In  order  to  confirm  accuracy  of  the  automated  system,  results  from  the  system  are  compared  to  a 
standard;  results  accomplished  manually  by  an  expert. 

A  comparison  of  results  for  a  2-minute  time  segment  between  the  present  automated  system  and  that 
obtained  manually  by  an  expert  are  represented  in  Figure  5. 
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Figure  4.  T-37  Pilot  Scan  Path  During  Instrument  Approach  (with  Reference  Markers) 
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Figure  5.  Comparison  of  Expert  and  Automated  Analysis  Showing  Percent  Time 
On  Instruments  In  T-37  Cockpit  For  One  Approach.  The  correlation  coefficient 
(r)  between  expert  manual  and  the  automated  analysis  system  is  equal  to 
0-9983. 
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INITIAL  APPLICATIONS  AND  RESULTS 


Initial  applications  for  the  current  study  were  decided  upon  in  a  joint  meeting  between  Armstrong 
Laboratory,  Human  Systems  Center  (HSC),  and  Headquarters  Air  Education  and  Training  Command  (HQ 
AETC).  Representatives  from  HSC  worked  with  HQ  AETC  to  determine  deficiencies  and  needs  that  had 
been  identified  in  the  training  environment. 

The  scientists  from  the  laboratory  then  worked  with  HQ  AETC  to  determine  where  the  technology 
would  have  the  greatest  impact  on  training  based  on  its  capabilities.  The  T-37  and  the  F-16  were 
subsequently  identified  as  areas  of  interest  for  data  collection.  The  study  was  designed  in  two  phases,  the 
first  being  a  “proof-  of-concepf  phase  using  IPs,  and  the  second  phase  designated  for  working  with 
students  in  an  actual  training  environment. 

During  Phase  I,  comments  from  the  pilots  were  gathered  through  questionnaires  regarding  general  use 
and  comfort  and  overall  response  to  the  eye  tracking  system.  Whenever  possible,  these  suggestions 
were  incorporated  into  the  system  as  it  evolved. 

The  first  application  chosen  for  the  portable  eye  tracking  and  analysis  systems  involved  precision  and 
nonprecision  instrument  approaches  performed  by  T-37  IPs  and  student  IPs  in  a  T-37  motion  base 
simulator  at  Randolph  AFB,  TX.  Eye  movement  data  were  recorded  from  16  pilots  while  performing  three 
instrument  approaches:  an  instrument  landing  system  (ILS),  a  localizer  (LOC),  and  a  VHF  omnidirectional 
range  (VOR).  Each  session  lasted  approximately  50  minutes.  Calibration  of  the  eye  tracking  system  was 
performed  at  the  onset  of  the  experiment.  Setup  and  calibration  were  typically  accomplished  in  four 
minutes  or  less.  Each  mission  was  flown  twice,  once  without  running  commentary,  and  the  second  time, 
with  the  pilot  providing  verbal  commentary  describing  the  procedures  necessary  to  perform  the  tasks  at 
hand.  We  plan  to  compare  the  verbal  commentary  to  the  actual  eye  position  data  in  order  to  determine 
the  accuracy  of  the  verbal  description  to  the  pilot's  line-of-sight.  A  histogram  displaying  eye  position  data 
for  one  approach  is  shown  in  Figure  5. 

The  primary  goal  of  the  F-16  portion  of  Phase  I  was  to  obtain  feedback,  from  pilots  who  are  current  in 
the  weapons  system,  on  the  use  of  the  eye-tracking  system  as  a  potential  training  device.  After  several 
demonstrations,  we  determined  the  best  use  of  the  system  at  point  was  in  the  creation  of  training  tapes 
demonstrating  effective  scanning  behavior  for  various  complex  scenarios.  The  scenarios  identified  for 
taping  in  the  F-16  ACES  at  Luke  AFB  include  basic  fighter  maneuver  (BFM)  setups  and  basic  air 
intercepts,  ending  with  air  combat  maneuvers  (ACM).  In  the  F-16  LANTIRN  facility,  also  at  Luke  AFB,  the 
scenarios  included  GLIB  II,  Terrain  Following  Radar  (TFR)  Letdown,  Laser  Guided  Bomb  (LGB)  Loft, 
Maverick  Attack,  and  Forward  Looking  Infrared  (FLIR)  direct  attack.  Setup  and  calibration  of  the  eye¬ 
tracking  system  was  accomplished  in  less  than  four  minutes  for  most  of  the  pilots.  Eight  F-16  LANTIRN 
IPs  and  F-16  IPs  participated  in  this  portion  of  Phase  I.  Training  tapes  generated  from  this  phase  are  now 
being  used  at  Luke  AFB  for  the  F-16  scenarios. 

As  a  result  of  our  work  at  both  Randolph  AFB  and  Luke  AFB,  several  improvements  were  made  to  the 
portable  system.  Improvements  include  an  integrated  communications  system,  added  headband  comfort, 
reduced  setup  and  calibration  times,  improved  accuracy,  remote  VCR  control  capabilities,  compatibility 
with  collimated  HUD  imagery,  and  improved  recording  and  playback  quality.  We  are  further  working  on 
schemes  that  will  allow  use  of  the  automated  analysis  system  with  the  F-16.  The  results  of  Phase  I  for  the 
T-37  and  F-16  studies  were  briefed  to  AETC  and  the  applications  for  Phase  II  are  being  determined. 

CONCLUSIONS 

Portable  eye  tracking  and  companion  analysis  systems  have  been  developed  which  allow  accurate 
measurement  of  eye  position.  The  eye  tracking  system  combines  eye  position  data  with  imagery  from  a 
head-mounted  scene  camera.  An  image-based  analysis  system  then  extracts  head  position  based  on 
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identification  of  reference  markers  providing  head  position  information.  A  measure  of  head  and  eye 
position  are  combined  to  provide  direction  of  gaze  information. 

The  portable  eye-tracking  system  can  be  employed  for  a  variety  of  applications  and  purposes.  We 
have  demonstrated  the  use  of  the  system  in  several  scenarios  and  propose  its  use  for  improving  training 
effectiveness.  The  portable  system  can  also  be  used  as  a  device  for  distinguishing  strategy  differences 
between  "expert”  and  "novice,”  or  less  skilled  individuals.  Efforts  are  also  presently  undenvay  to  develop 
improved  references  for  use  with  the  automated  system  as  well  as  to  improve  system  accuracy. 

The  purpose  of  Phase  II,  which  should  be  complete  by  the  end  of  1997,  is  to  identify  the  effectiveness 
of  eye  tracking  in  training  pilots  to  fly  complex  maneuvers.  Through  all  of  the  tasks  accomplished,  from 
the  F-15  situational  awareness  effort  to  the  ongoing  research  effort  with  AETC,  we  have  seen  that  eye¬ 
tracking  technologies  can  be  used  to  maximize  training  potential.  The  portable  system  that  is  continually 
being  developed  through  the  joint  efforts  of  AL,  AETC,  HSC,  and  El  Mar  has  been  designed  to  maximize 
efficiency  and  accuracy,  always  considering  the  needs  of  the  operational  pilot. 
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